Instructors may present pedagogical “items”—e.g., questions posed to participants in, for example, a classroom environment—to teach or otherwise instruct students. It is challenging, however, to develop effective items, and the challenge extends beyond thinking of good questions to ask. The larger problem is that it is difficult to know ex ante how an item will function among the participants that respond to it—e.g., how difficult an item will be, how distracting the “distractors” (incorrect response options) in a multiple-choice item will be, etc. The most useful in-class questions in peer instruction and peer learning are those which discriminate among the students in their degree of knowledge, and which reveal their misunderstandings. For peer instruction, ideally only about half of the students will have the right answer initially, and probably fewer will understand why it is the right answer.
For these reasons, the items that make up large-scale, high-stakes standardized tests are extensively pretested so that poorly functioning items (e.g., those with too high or low a percentage of correct answers) can be revised or discarded. However, instructors are confronted with inadequacy of their data sets and they rarely (if ever) have the resources to do this type of pretesting for their own items. In particular, instructors cannot influence the result of the items, which leads to biased results. An additional problem is that pretesting among students in one's own class typically makes it impossible to keep the class discussion fresh and to obtain accurate measurements of student learning, yet testing in other classes is not feasible for most instructors. It is desirable to replicate randomized participant data sets as closely as possible by obtaining treated and control group of participants with similar covariate distributions and characteristics.
Consequently, there is a need for an approach that can provide easy and fast guidance to instructors on how to best validate the items and obtain useful results in class.